The Minimum Information Principle in Learning and Neural Data Analysis

As science enters the 21st century, the study of complex systems is increasingly gaining importance. Complex systems are characterized by an extremely large number of units (many millions) which cooperate to produce complex observable behavior. Perhaps the most striking example of such a system is the human brain, a network of 1010 nerve cells whose simultaneous activity results in all human functions, from locomotion to emotion. Understanding the brain involves several extremely difficult paradigmatic and experimental challenges. At the current point of time, it is impossible to observe the activity of more than a few hundred cells simultaneously. Furthermore, even if one could obtain such a ”Brain TV” which reports the simultaneous activity of all nerve cells in the brain, it is not clear that such a device would lead to a satisfactory explanation as to how the brain operates. These conceptual difficulties are not unique to brain research. They are manifested in the study of other complex systems such as biological networks (i.e., metabolic pathways), weather forecasting, and huge document databases such as the world wide web. Machine learning is an important field of research which studies fundamental theoretic and algorithmic aspects of extracting rules from empirical measurements of such systems. We make frequent use of machine learning concepts throughout the dissertation. It is clear that any empirical measurement of complex systems is bound to be partial, in the sense that only a subset of its units may be measured at a given time 1. Furthermore, experimental procedures typically limit the duration for which one can observe a given unit 2. We are thus faced with the following fundamental question: what can one say about a system, given such a set of partial observations? An immediate extension of this question is how to choose the observations which would be most beneficial for studying a system. These two questions, and their various extensions are the focus of this dissertation. One of the central formal tools in the current thesis is Information Theory, introduced by Claude Shannon in the 1940’s as a comprehensive mathematical theory of Perhaps with the exception of the world wide web. This state of affairs may be considerably enhanced in the future, with the improvement of chronically implanted electrodes [39]. communications. Information Theory allows one to quantify information transmission in systems and is thus an attractive tool for studying information processing systems like the brain. Substantial literature exists on estimating information between neural activity and the external wold, in an effort to understand how the world is encoded in cortical activity. However, information theoretic methods also face two difficulties when applied to complex system analysis: one is the partial measurement problem described above, and the other is the need to understand which properties of the system are important for its function (for example: what is the information in single neuron responses as compared to information in neuronal correlations). The current dissertation deals with these methodological issues by designing tools for measuring information under partial measurement and in specific properties of the system. The new Minimum Mutual Information (MinMI) principle , developed here, quantifies information in these scenarios by considering all possible systems with a given property, and returning the minimum information in this set of systems. The resulting number captures the information in the given property, since systems with higher information necessarily contain additional, information enhancing properties. Furthermore, it is a lower bound on the information in the system whose partial properties were measured. The first chapter of the thesis covers previous approaches to the problem of inference from partial measurements. This problem was previously addressed in the framework of the Maximum Entropy (MaxEnt) principle, developed by Maxwell and Jaynes [73], and more recently in the machine learning [34], and neural coding [122] literature. MaxEnt is similar to MinMI, in that it addresses all systems with a given property. However, unlike MinMI, MaxEnt returns the system that maximizes entropy, which is an information theoretic measure of uncertainty. The MinMI principle is the natural extension of MaxEnt methods to information processing systems, since it is targeted at information measurement. The relation between these two principles is discussed in Chapters 2 and 3. Chapter 1 also covers basic concepts in information theory and geometry of distributions, which will be used throughout the thesis. The MinMI principle is presented in Chapter 2. The general form of its solution is derived, and several algorithms are given for calculating its parameters. The algorithms are based on geometric projections of distributions (I-projections), and con-